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ABSTRACT 

An assessment of the review process for selection of 
documents for inclusion in the Clearinghouse on Tests, Measurement, 
and Evaluation of the Educational Research Information Center 
v ERIC/TM) is presented. The review process is used to cull irrelevant 
or technically flawed material from the over 1,500 acquired 
documents. The clearinghouse, which is one of the 16 specialized 
clearinghouses in the ERIC system, acquires, selects, processes, and 
announces approximately 750 published articles and 800 unpublished 
documents each year. To date, approximately 14% of the reviewed 
documents have been rejected and 38% transferred to other 
clearinghouses. Major review criteria include -elevance and 
timeliness, methodology, and effectiveness of presentation. This 
study was conducted during November and December of 1938; 17 
measurement specialists acted as external reviewers of the ERIC/TM 
review process. Thirty documents were selected from among those 
reviewed previously by the clearinghouse; 20 of these were randomly 
selected from a group of documents that had been accepted since June 
1988. In addition, 10 documents rejected since Auguot 1988 were 
selected. The agreement rates between ERIC/TM and the external 
reviewers suggest that ERIC/TM is reviewing and accepting documents 
in a manner consistent with the expectations of the educational 
community. The ERIC/TM Document Review Form and an outline of review 
criteria are appended. (TJH) 
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Introduction 

As Rudner and Wise (1989) point out, vigilance in selecting documents for inclusion 
into the ERIC database is a major task of the Educational Research Information 
Center's Clearinghouse on Tests, Measurement, and Evaluation (ERIC/TM). 
ERIC/TM's review process is the method used to "weed out irrelevant or technically 
flawed material" from over 1,500 acquired documents (Rudner and Wise, 1988, p. 
15). The process must be continually n.onitored by ERIC users to ensure its 
continued utility. This study addresses this need for monitoring by asking: "How 
can ERIC/TM better serve its audience in terms of better selection of documents 
for inclusion into the ERIC database?" 

This paper reports the results of ERIC/TM's evaluation of its document review 
process. The study was conducted during November a.id December of 1988. This 
report details: 

• what ERIC/TM is 

• what ERIC/TM's document review procedures are 

• the method of this evaluation 

• the results of this evaluation 

• conclusions from the evaluation 

This evaluation replicates and extends an earlier evaluation of ERIC/TM's review 
process (Eissenberg & Rudner, 1988). Eissenberg and Rudner reported satisfactory 
results, but noted that ERIC/TM should be paying closer attention to methodology 
when judging documents and that documents not directly related to education should 
still be considered for inclusion in the ERIC database. 
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What Is ERIC/TM? 



ERIC/TM is one of the 16 specialized clearinghouses in the ERIC system. ERIC 
clearinghouses are responsible for building a database of "fugitive" literature, a 
database of published articles, and synthesizing literature within their field of 
interest, 

ERIC/TM acquires, selects, abstracts, processes, and announces approximately 750 
published articles and 800 unpublished documents about to all aspects of educational 
testing, evaluation, measurement, and learning theory. 

Articles and manuscripts selected for inclusion in the ERIC system have high 
visibility in the educational community. The full text of accepted non-journal 
documents are archived on microfiche and made available more than 800 locations. 
Abstracts of all accepted manuscripts can be searched through major online 
information retrieval systems, such as Dialog, BRS, and SDC, and through CD-ROM 
systems, such as OCLC, and SilverPlatter. Abstracts are also published in Resources 
in Education or Current Index to Journals in Education, which are available in more 
than 2,500 libraries and information centers. 

In developing the testing portion of the fugitive literature database, ERIC/TM 
acquires and reviews approximately 1500 unpublished articles annually. Documents 
outside the TM scope of interest are forwarded to the appropriate clearinghouse. 
ERIC/TM reviews documents within its scope, following the procedures described in 
the next section. To date, approximately 14% of the reviewed document* have been 
rejected and 38% transferred to other clearinghouses (Hannaman, 1989). 

ERIC/TNi undertook a self-evaluation in order to improve its services to the 
measurement and educational communities. ERIC/TM hopes that this evaluation, as 
well as future evaluations, will increase the already high quality of the ERIC 
database. 



What Are ERIC/TNTs Review Procedures? 



After ERIC/TP/I receives and catalogs documents, the documents are handed to the 
Review Coordintor. The primary function of the Review Coordinator is to decide 
if ERIQ7M should 

• accept and continue to process the document, 

• reject the document, or 

• transfer the document to another clearinghouse 

This important decision is based on several clearly defined review criteria. These 
criteria are: 

• relevance and timeliness 

• methodology 

• effectiveness of presentation 

Relevance and Timeliness: ERIC/TM only accepts into the ERIC database documents 
that contribute new information to the field of educational measurement. If a 
particular document contributes to another field of education, ERIC/TM transfers it 
to the appropriate clearinghouse. When documents clearly do not contribute, 
ERIC/TM rejects them. 

When judging a document's relevance and timeliness, we ask questions such as 

Does the document address a concern of the measurement community? 

Is the document on the edge of measurement research? 

Does the document discuss a past measurement issue in a new light? 

Methodology: We judge the methodology described in a document by examining the 
document's objectives, 

research procedures, and 

conclusions and recommendations. 
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ERIC7TM often judges research reports based on all three of these criteria. Other 
documents - for example, position papers - may contribute heavily in only one or 
two of these criteria. 

Some questions that ERIC/TM asks when it reviews a document's contribution are: 

Does the document meet its objectives? 

Are the document's research procedures sound? 

Are the conclusions justified? 

Effectiveness of Presentation: ERIC/TM also judges documents based on their 
effectiveness of presentation. This criterion includes: 

• clarity of language 

• writing style 

• audience 

• cohesiveness 

WhUe presentation quality may vaiy, ERIC/TM strives to accept those documents 
that are clearly written and easy to follow. 

Some questions ERIQTM asks when judging the effectiveness of a document's 
presentation are: 

Are terms clearly defined where necessary? 

Are explanations in the document easy to understand? 

Does the document address its audience? 

Does the document flow logically? 

Other Criteria Of course, these three major criteria are not the only criteria 
ERIC/TM uses when reviewing documents. It also examines each document for 



• new applications of knowledge 

• authority of the author 

• thoroughness of reporting 

• responsiveness to current priorities 

• technical cc ^derations 

All of these criteria help ERIG/TM to achieve its goal of accepting only documents 
that benefit the many users of the ERIC system. 

What Was the Method of ERIQTM's Evaluation? 

In November and December of 1988, ERIC/TM contacted 17 measurement specialists 
who agreed to help with the evaluation of the review process by being external 
reviewers. These 17 external reviewers included test designers, researchers, policy- 
makers, writers, publishers, and practitioners in the field of educational 
measurement The external reviewers reviewed documents that ERIC/TM had 
already decided to accept or reject. 

Because the evaluation was of the review process, ERIC/TM selected 30 documents 
from the documents that it had already reviewed. Of these 30 documents, 20 were 
randomly selected from a larger group of documents that had previously been 
accepted since June, 1988. ERIC/TM also randomly selected 10 documents from all 
documents that had been rejected since August, 1988. Most documents (62%) that 
came to ERIC/TM in 1988 were either accepted or rejected. (Hannaman, 1989). 
ERIC/TM did not select documents for this evaluation that had been transferred to 
other ERIC clearinghouses. 

Once these 30 documents were selected, ERIC/TM randomly assigned each external 
reviewer between nine and 13 documents to review. Documents were assigned so 
that if every reviewer responded, each document would be reviewed by at least six 
external reviewers. 
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ERIC/TM sent each external reviewer 



• the documents to review 

• 12 document review forms (see Appendix A) 

• a brief explanation of ERIC/TM review criteria (see Appendix B) 

ERIC/TM included document review forms to help reviewers with their decision. 
These forms contain a rating scale for each of the three major selection criteria. 
Also, the document review forms provided space for reviewer comments, enabling 
ERIC/TM to better understand reviewer decisions. ERIC/TM explained that 
reviewers would be making only "accept" or "reject" decisions. The external 
reviewers were unaware of any previous review decision made by ERIC/TM. 



What Were the Results of the Evaluation? 

Of the 17 measurement specialists who agreed to participate, 11 responded with the 
results of their review. These results are reported in two levels of analysis 

• Inter-rater agreement 

• Document agreement and disagreement 

Inter-rater Agreement: The individual results of the external reviewers are 
compared to ERIC/TM's review in Table 1. On the next page, Table 1 shows a 66% 
agreement rate between ERIC/TM and the external reviewers as a group. 
Individually, agreement rates ranged from a low of 34% to a high of 86%. 
Eissenberg and Rudner (1988) reported a 70% agreement rate, with a range of 63% 
to 88%. 
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Table 1: Comparison of internal ERIC/TM review decisions with 
decisions made by external reviewers. 



Agreement 
ERIC/TM Decision Accept Reject 
Ext. Reviewer Accept Reject 



Decision Type 

Disagreement 
Accept Reject 
Reject Accept 



% Agreement 



1 

2 
3 
4 
5 
6 
7 
8 
9 

10 
11 



Totals 



5 
6 
5 
5 
7 
5 
1 
7 
5 
4 
4 



54 



2 
2 
4 
3 
1 
3 
2 
2 
2 
2 
2 



25 



1 


2 


70% 


t 


1 


80% 


2 


2 


69% 


2 


1 


72% 


1 


3 


67% 


3 


1 


67% 


5 


1 


34% 


1 


2- 


75% 


3 


3 


54% 


4 


2 


50% 


0 


1 


86% 


23 


18 


66% 



Document Agreement and Disagreement: On the next page, Table 2 shows the 
agreement rates for individual documents that ERIC/TM accepted. This table shows 
that there were only two instances in which all of the external reviewers of a 
document disagreed with ERIQTM's decision to accept that document (numbers 19 
and 20). One of these two documents was reviewed by only one external reviewer. 
There were five instances in which all of the external reviewers of a document 
agreed with ERIC/TM's decision to accept the document (numbers 2, 3, 6, 10, and 

\6). 
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Tabic 2: Comparison of ERIC/TM's accepted documents with external review 
decisions 



Short Titles 
of Documents 

ERIG/TM Accepted Reviewer 


External Reviewer Derisions 
1 2 3 


4 


5 


6 


1. Nafl high school grad. reqs. 


Arrant 




Kcjeci 


Accept 






2. GenT cognitive operations 


A rrpnt 


Accept 










3. Comparison of models in CAT 


A crj*nt 




/\ccepi 








4. Evaluation of PET 


Acrmt 


AtfYMtt 1 


/\cccpi 


Accept 


Accept 


Reject 


5. impa^ of Governor's school 


Ar/vnt 


A <*l*#»T>fr 


Reject 


Accept 






6. Attitudes about ed. test formats 


Accept 


Ar/*i»nt 
nvtcui 




Accept 


Accept 




7. Cntical thinking & curr. design 




k eject 


Accept 


Accept 


Accept 


Accept 


8. School and teacher effectiveness 


Arrant 




IxCJCCl 








9. Eval. of SABEn demo, school 


Accept 


Accept 


Reject 








10. Indicators of literacy 




Accept 


Accept 


Accept 






11. Rel. & val. of behav. inventory 


Reject 


Accept 


Reject 


Accept 


Reject 




12. Leadership assessment instrument 


Accept 


Reject 


Accept 


Accept 






13. Measunng health knowledge 


Accept 


Accept 


Accept 


Reject 


Reject 


Reject 


14. ETV formative evaluation 


Reject 


Accept 


Reject 


Accept 


Reject 




15. Modifying Stouf s procedure 


Reject 


Reject 


Accept 








16. Evaluating no pass-no play 


Accept 


Accept 


Reject 


Accept 


Accept 


Accept 


17. Evaluating grade retention 


Accept 


Reject 










18. Comparing test-taking strategies 


Accept 


Accept 


Accept 


Accept 






19. Assessing stud ; skills behavior 


Reject 












20. Dissatisfaction theory 


Reject 


Reject 


Reject 









On the next page, Figure 1 shows the percentage of external reviewers who 
disagreed with ERIC/TM's decision to accept a document - the percent of 
reviewers who would have rejected a document ERIQTM accepted. As the figure 
shows, only three documents out of 20 had a disagreement rate higher than 66%. 
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Figure 1: Percent of external reviewer "-ERIC/TM disagreement 
regarding document* that FRIC/TM accepted. 




1 2 3 4 6 6 7 8 9 10 11 12 13 14 16 16 17 18 1920 
Piwiwm minbw 



Figure 2: Percent of external reviewe r C RIC/TM disagreement 
regarding documente that ERIC/TM rejected. 
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Tabic 3 shows Ac agreement rates for individual documents that ERIC/TM rejected. 
This table shows that there were no instances in which all external reviewers of a 
document disagreed with ERIOTM's decision to reject that document. In contrast, 
there were thrc- instances in which all external reviewers of a document agreed 
with ERIC/TNTs ». ..sion to reject the document. 



Tabic 3: Comparison of ERIC/TNTs rejected documents with external review 
decisions 



Short TWes 
of Documents 



EamiiLBgds8gJ2fiaapjB 



ERIC/TM Rejected Reviewer 


1 


2 


3 


4 


5 


21. Chicago test scores, 1986 


Accept 


Reject 


Accept 






22. Cooperative research in schools 


Reject 


Reject 


Reiect 






23. Dev. military proficiency 


Reject 


Accept 


Reject 


Reject 




24. EvaL of PhysJSd. pilot program 


Reject 


Accept 


Accept 


Accept 


Reject 


25. Product design and curr. implera. 


Reject 


Reject 


Accept 


Accept 


Accept 


26. Pred. valid, of ACT assessment 


Accept 


Accept 


Accept 


Reject 


Accept 


27. Depre&, aix, and self-efficacy 


Reject 


Reject 


Reject 


Reject 




28. Predictability of GPA . \iMA 


Accept 


Accept 


Accept 


Reject 




29. Computation of GCV functions 


Reject 


Accept 








30. Washington statewide assessment 


Reject 


Reject 


Reject 


Reject 


Reject 



Figure 2 displays the percent of external reviewers who disagreed with ERIC/TNTs 
decision to reject a document As the Figure shows, 4 of the 10 documents had a 
disagreement rate higher than 66%. 
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Discussion and Conclusions 



The Inter-rater Agreement: One of ERIQTM's goals is to accept documente into 
the database that interest a wide audience within the educational community, while 
also not overloading the database with documents that are of little value to this 
community. ERIQTM views the overall agreement rat? of 66 percent as an 
indication that it is achieving this goal. This relatively high rate of agreement 
suggests that ERIQTM's review process is reliable and appropriate. This conclusion 
is especially noteworthy considering the varied specialties of the external reviewers 
and the similar agreement rate reported by Eissenberg and Rudner (1988). 

The Individual Document Agreement Rate: The agreement rates between ERICVTM 
and the external reviewers suggest that ERICVTM is reviewing and accepting 
documents in a manner that is consistent with the expectations of the educational 
community. However, ERICTM also examined some of the critical incidents of 
disagreement regarding individual documents to identify where it can improve the 
review process. 

In this evaluation, a critical incident of disagreement occurs when more than 66% of 
the external reviewers disagreed with ERIQTM's decision to accept or reject a 
particular document Eissenberg and Rudner (1988) defined a critical incident of 
disagreement as a document that provoked 100% disagreement and reported eight 
such incidents. This stricter definition can be attributed to the fact that in most 
cases (28 of 40) only one or two external reviewers examined a document In the 
present study, there were seven cases in which 67% (or more) of the reviewers 
disagreed with ERIQTM's decision about a document, and in most cases (26 of 30), 
thr~. or more external reviewers examined a document 

In three cases, at least 67% of the external reviewers rejected documents that 
ERIQTM had accepted, though one of these documents was reviewed only by a 
single external reviewer, and thus there is little basis for discussion. In four cases, 
at least 67% of the reviewers accepted documents that ERIQTM had rejected. 
Several conclusions can be drawn from examining the individual cases presented by 
these two types of disagreement. 
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ERIC/TM accepts, external reviewers reject: There were three cases in which at 
least 67% of the external reviewers disagreed with ERIGTM's decision to accept a 
document. One of these three cases, document number 19, had only one external 
reviewer. Since one external reviewer is not a meaningful two-thirds majority, we 
will not discuss this document further. However, the oth^r two cases were 
reviewed by three external reviewers, and thus deserve closer examination. 

Document 20 was reviewed by three external reviewers, and they all would have 
rejected the document One of the external reviewers described the document as "a 
study of a particular hypothesis in the 'sociology of education' -- why school board 
members are defeated in elections." As such, though the methodology and 
presentation were adequate, the document was not relevant to ERIQTM. This 
opinion was supported by the other two reviewers who commented "Irrelevant to 
ERIC/TM" and "Wrong ERIC." 

When ERIC/TM accepted this document, it was because the document was relevant 
to the field of education and to the educational community. Furthermore, the 
document represented an attempt to extend a statistical model related to school 
board election results. Since no clearinghouse seemed to have a direct claim on 
this document and since the document might appeal to the greater ERIC audience as 
well as to some educational statisticians, ERIC/TM chose to accept it. The 
unanimous decision of our evaluators suggested that we may have overstepped our 
scope of interest (but see the discussion of document 25 below for a counterpoint). 

The case of the other document that our reviewers rejected is not so clear cut. 
The document is a technical discussion of a modification of a procedure for 
assessing latent trait unidimensionality. The three reviewers commented: 

Unless this sort of paper is carefully reviewed by orie familiar with the 
technical notation it can lead to incorrect applications. Therefore, I 
mink it should be submitted for publication to a technical journal such as 
Pwchometrika rather than to be included in ERIC. It is not likely to be 
of interest or useful to ERIC users. (Reject) 

Terrible presentation. It reads like a TV writers stt-eotypc of computer 
output (Reject) Y 
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Although in an almost outline form, the paper gets to the substance of 
the matter and is clear to follow. (Accept) 

Because of the detailed statistical content, this document was eventually reviewed 
by ERIC/TM's director. Most technical documents, such as number 29 (rejected), 
receive extra attention precisely for the reason noted by the first reviewer. 
However, statistical documents are of interest to some members of the measurement 
community and, thus, will interest some of ERIC/TM's audience. Therefore, 
statistical documents that contribute significantly to the field of educational 
measurement wih be included in the ERIC database. At least one of our .eviewers 
(the third) and possibly another (the first) believe that this document makes such a 
contribution. 

ERIC/TM rejects, external reviewers accept: At least 67% of the eternal reviewers 
decided that ERIC/TM should have accepted four documents that it rejected. The 
first of these, document 20, was a report of Fall, 1986 test scores for selected high 
schools in the Chicago Public Schools. One reviewer noted the limited interest of 
this document, but felt that it should be accepted because it described results by 
school. Another reviewer commented that ERIC seemed a fitting place for school 
district test reports. 

ERIC/TM receives many such school district reports, more than would be of use to 
most ERIC users. Ratner than not accepting any of tfiese reports, ERIC/TM tries 
to accept reports that contain meaningful descriptive text, not table after table. 
We believe that our audience is better served by reports that discuss results in 
some depth. While this document does contain some descriptive text, ERIC/TM 
agrees with tL. t'ocunent's third reviewer who found the document inadequate in 
terms of relevance and effectiveness of presentation. When a group of district 
scores are reported in an exceptional manner, in terms of presentation, conclusions, 
and summaries, ERIC/TM accepts such a report, interestingly, Eissenberg and 
Rudner (1988) reported that a report of state scores that ERIC/TM accepted was 
rejected by both of its external reviewers. 

Another critical incident of disagreement, document 25, is a description of 
curriculum implementation in The Netherlands. While two reviewers agreed with 

13 
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ERIC/TM and rejected the document, four chose to accept it. While one reviewer 
found the document a "first class overview," the comments of the other three reveal 
that their choice was not so clear: 

My first reaction was to conclude that this report ... did not belong to 
ERIC/TM. Then I found there is no ERIC center that concentrates on 
general curriculum issues . . . 

Marginal call; after a good brief review and a strong argument for 
observational data, the author peters out; he doesn f t have many results to 
report, and he leaves one big question . . . unanswered except for 
speculation. 

Does this belong to the TM clearinghouse's domain? I found it very hard 
to follow . . . However, the paper may be useful to those "in the know." 

This document was not an easy one to review. Its international perspective and 
undeniable educational content make it a strong candidate for inclusion in the ERIC 
system, though possibly not as an ERIC/TM docume nt Since our reviewers felt 
that it should not have been rejected, ERIC/TM might have better served ERIC 
users by transferring the document to another clearinghouse. In the future, 
ERIC/TM will make a stronger effort not to reject a document from further 
consideration (by another clearinghouse) unless the document will clearly not serve 
the interests of the vast majority of ERIC users. 

The next rejected document with at least 67% disagreement from external reviewers 
was a three-page discussion of the predictive validity of the ACT for various ethnic 
groups. This discussion was written in 1981. All the external reviewers noted this 
document's relevance to ERIQTM's scope, but only one reviewer noted the age of 
the document. Timeliness is a major criterion that all clearinghouses consider, and 
ERIC/TM is no exception. This document was rejected, in part, because it is seven 
years old. 

Age was not the only criteria that contributed to ERIQTM's rejection however. 
Another consideration is document length. The ERIC Processing Manual, the 
document that describes the rules and guidelines under which an ERIC Clearinghouse 
operates, notes that: 

14 
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Short documents can create user problems, and their entiy into . . . 
[ERIC] should be kept to a minimum . . . short documents selected for 
the ERIC system must be scrutinized with greater than ordinary care. 
They must be documents of high quality (U.S. Dept. of Ed., 1980, p. IJI- 

This attention to document length is a criterion of the ERIC system unknown to our 
reviewers. Given the length of the document, then, ERIC/TM's rejection seened 
acceptable. Based on our external reviewers' 83% disagreement with ERIC/TM's 
decision, ERIC/TM may serve its audience best by paying closer attention to the 
relevance of a document, and then weighing the relevance against the timeliness and 
the length. As one reviewer commented, "I'd accept [this document] because of 
[the] importance of ACT assessment and [the] continuing timeliness of the question 
of differential predictive validity." 

The last document to be considered, number 28, was another three-page document 
that discussed material of a high relevance for the external reviewers and thus 
ERIC/TM's audience. The document discusses the stability of grade point averages 
at the United States Military Academy. Though this document was originally 
rejected because it seemed to lack sufficient content, our reviewers chose to accept 
ii. Again, ERIQTM might better serve its audience by paying more attention to 
document content and less to document content and document size. 

Summary: Given ERIC/TM's audience of "researchers, state and federal 
policymakers, local decisionmakers, practitioners, curriculum and test developers, and 
consumers" (Rudner & Wise, 1989, p. 12), ERIC/TM will not always be able to please 
every one when it applies a selective review process. However, the results of this 
and earlier evaluations of the selection process indicate that, generally, ERIC/TM is 
pleasing most of its audience most of the time. 

Of course, improvements can be made in ERIC/TM's selection process. This 
evaluation shows that ERIC/TM should be sure to 

• try to transfer quality documents that are not clearly in any 
clearinghouse's scope, rather than rejecting these documents 

• First judge the relevance of a document and then weigh that judgment 
against the document's timeliness or length 

15 
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The interests and needs of ERIC/TM's audience will evolve and grow, just as the 
various fields in education evolve and grow. ERIQTM will need to continually 
evaluate its review process in order to ensure that it can serve the changing needs 
of its audiences. 
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Appendix A; ERIC/TM Document Review Form 




ERIC/TM Document Review 

Reviewer 



Date: 



Author: 



Short Title: " 
Review Criteria 

Good Adequate Inadequate 

1. Relevance/timeliness 

2. Methodology 

3. Effectiveness of 

presentation 

4. Other (optional) 
Specify: 



Comments: 




Recommendation: 

Accept Reject 
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ERIC/TM Document Review Criteria 

Every document acquired by the ERIC system is carefully reviewed by an ERIC 
clearinghouse for technical and content merit. Judging document content is the 
most important and demanding part of this review process. 

At the ERIC Clearinghouse on Tests, Measurement, and Evalutaion (ERIQTM), we 
accept and process only documents that have a high potential for making a 
contribution to the field of educational measurement. We judge documents' 
contribution to educational measurement by examining their 

• relevance and timeliness 

• methodology 

• effectiveness of presentation 

There are two main reasons for following a careful review procedure: 

1. The ERIC database is most useful to its users if it contains only the most 
relevant educational research 

1 The ERIC database is most cost effective when clearinghouses process only the 
most relevant documents. 

Relevance and Timeliness 

0 

ERIC/TM accepts into the ERIC database only documents that contribute new 
infoimation to the field of educational measurement. If the document contributes to 
another field of education, we transfer the document to the appropriate 
clearinghouse. When documents clearly do not make a contribution, ERIC/TM 
rejects them. 
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When judging a document's relevance and timeliness we ask questions such as 

Does the document address a concern of the measurement community? 

Is the document on the edge of measurement research? 

Does the document discuss a past measurement issue in a new light? 

Methodology 

We judge the methodology described in a document by examining the document's 

• objectives 

• research procedures 

• conclusions and recommendations 

We can often judge research reports based on all three of these criteria. Other 
documents, for example position papers, may contribute heavily in only one or two 
of these criteria. 

Some questions that we ask when we review a document's contribution are: 

Does the document meet its objectives? 

Are the document's research procedures sound? 

Are the conclusions justified? 



Effectiveness of Presentation 

We also judge documents based on their effectiveness of presentation. This 
criterion includes: 

• clarity of language 

• writing style 

• audience 

• cohesiveness 
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While presentation quality may vary, we strive to accept those that are clearly 
written and easy to follow. 

Some questions we ask when we judge the effectiveness of a document's 
presentation are: 



Are terms clearly defined where necessary? 

Are explanations in the document easy to understand? 

Does the document address its audience? 

Does the document flow logicaily? 



Do We Use Other Criteria to Review Documents? 

Of course, car three major criteria are not the only criteria we use when we 
review documents. We also examine each document for 



new applications of knowledge 
authority of the author 
thoroughness of reporting 
resoonsiveness to current priorities 
technical considerations 



All of these criteria help us to achieve our goal of accepting only documents that 
benefit the many users of the ERIC system. 



